
import Head from 'next/head'

<Head>
  <script>
    {
      `(function() {
         var _hmt = _hmt || [];
(function() {
  var hm = document.createElement("script");
  hm.src = "https://hm.baidu.com/hm.js?e60fb290e204e04c5cb6f79b0ac1e697";
  var s = document.getElementsByTagName("script")[0]; 
  s.parentNode.insertBefore(hm, s);
})();
       })();`
    }
  </script>
</Head>

![LangChain](https://pica.zhimg.com/50/v2-56e8bbb52aa271012541c1fe1ceb11a2_r.gif)






 代理基准：搜索+计算器
 [#](#agent-benchmarking-search-calculator "Permalink to this headline")
==================================================================================================================

>  代理基准：搜索+计算器 Agent Benchmarking: Search + Calculator 

在这里，我们将讨论如何在代理可以访问计算器和搜索工具的任务上对代理的性能进行基准测试。



 
强烈建议您在启用跟踪的情况下进行任何评估/基准测试。请参阅[here](https://langchain.readthedocs.io/en/latest/tracing) 了解什么是跟踪以及如何设置它。






```python
# Comment this out if you are NOT using tracing
import os
os.environ["LANGCHAIN_HANDLER"] = "langchain"

```







 加载数据 Loading the data
 [#](#loading-the-data "Permalink to this headline")
-----------------------------------------------------------------------



首先，让我们加载数据。
 







```python
from langchain.evaluation.loading import load_dataset
dataset = load_dataset("agent-search-calculator")

```








 设置链  Setting up a chain
 [#](#setting-up-a-chain "Permalink to this headline")
---------------------------------------------------------------------------



现在我们需要加载一个能够回答这些问题的代理。







```python
from langchain.llms import OpenAI
from langchain.chains import LLMMathChain
from langchain.agents import initialize_agent, Tool, load_tools
from langchain.agents import AgentType

tools = load_tools(['serpapi', 'llm-math'], llm=OpenAI(temperature=0))
agent = initialize_agent(tools, OpenAI(temperature=0), agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, verbose=True)

```








 预测 Make a prediction
 [#](#make-a-prediction "Permalink to this headline")
-------------------------------------------------------------------------



首先，我们可以一次预测一个数据点。在这种粒度级别上执行此操作允许use详细地探索输出，而且比在多个数据点上运行要便宜得多






```python
print(dataset[0]['question'])
agent.run(dataset[0]['question'])

```








 做很多预测 Make many predictions
 [#](#make-many-predictions "Permalink to this headline")
---------------------------------------------------------------------------------



在我们可以做出预测
 







```python
agent.run(dataset[4]['question'])

```










```python
predictions = []
predicted_dataset = []
error_dataset = []
for data in dataset:
    new_data = {"input": data["question"], "answer": data["answer"]}
    try:
        predictions.append(agent(new_data))
        predicted_dataset.append(new_data)
    except Exception as e:
        predictions.append({"output": str(e), \*\*new_data})
        error_dataset.append(new_data)

```








 评估性能 Evaluate performance
 [#](#evaluate-performance "Permalink to this headline")
-------------------------------------------------------------------------------



现在我们可以评估预测。我们能做的第一件事就是用眼睛看它们。






```python
predictions[0]

```






接下来，我们可以使用一个语言模型，以编程的方式给它们打分





```python
from langchain.evaluation.qa import QAEvalChain

```










```python
llm = OpenAI(temperature=0)
eval_chain = QAEvalChain.from_llm(llm)
graded_outputs = eval_chain.evaluate(dataset, predictions, question_key="question", prediction_key="output")

```




我们可以将分级输出添加到 `predictions` dict中，然后获得等级计数。


 







```python
for i, prediction in enumerate(predictions):
    prediction['grade'] = graded_outputs[i]['text']

```










```python
from collections import Counter
Counter([pred['grade'] for pred in predictions])

```






我们还可以过滤数据点，找出不正确的例子并查看它们。


```python
incorrect = [pred for pred in predictions if pred['grade'] == " INCORRECT"]

```










```python
incorrect

```








